I'm not sure a program-level argument is best -- isn't rather a subset of tasks that are ok to fail? Rather than mark them as if they were generated with a different tool, you could just mark using a glob, e.g.

[input_error/*]
can_fail = True

and then specify options on a per-task level as needed. This has the advantage that you always use the same executable for tasks intended to fail as those intended to pass.

What do you think?

Indeed, I originally implemented the can_fail option in the jobconfig file, rather than in the userconfig. However, I later felt the most appropriate place was the userconfig. This come from my understanding (or maybe lack of understanding...) of the philosophy of test code.

In particular, for Wannier90, we have two executables, but for each of them, depending on the options within the input file, a number of files might be generated, and one might want to parse more than one file (or different files for different types of tests).

Therefore, the way we implemented the use of test code in Wannier90 is the following: each "program" in userconfig does not only represent a binary to execute, but the combination:

executable + type of file to parse + parser + expected exit status

(in fact, the extract function needs to be specified in userconfig, so this seems sensible).
This approach was also needed I imagine because I think only one output file can be specified (and one extract function).

In the job config one just selects the specific combination above, and says the name of the output file. See e.g. our userconfig and jobconfig files.

With this approach, while putting can_fail in jobconfig would still work, we would duplicate information, because anyway we would need to use a different parser (to check the error code of the program, rather than the numeric outputs).

Do you think this makes sense? Or do you have a different suggestion for addressing our use case?